Night Owl 6/Night Owl's Shareware - PDSI-006 - Night Owl Corp (1990).iso/027a/clip5not.zip/S50003.TXT

Text File | 1990-10-08 | 17KB | 372 lines

CLIPPER 5.0 SUPPORT BULLETIN #3

BULLETIN REVISED: 04 Oct. 1990

PRODUCT: Clipper 5.0

AFFECTED VERSIONS: Rev. 1.00 - 1.03

SUBJECT: Clipper 5.0 runtime memory management


This document provides information on the Clipper 5.0 runtime
memory management system.

Note: this document makes several references to the CLIPPER
environment variable.  For general information on how to
configure the runtime environment, refer to the Clipper 5.0
"Programming and Utilities Guide."


VIRTUAL MEMORY MANAGER (VMM)

  "Virtual memory" is a generic term for hardware or software
  techniques that allow a limited amount of "real" memory to
  emulate a much larger "virtual" memory.

  The Clipper 5.0 Virtual Memory Manager (VMM) is implemented in
  software.  The system runs in real mode on any 8086-compatible
  processor; it does not directly use the protected mode virtual
  memory capabilities of the Intel 286 and 386 processors.

  The VMM is a "segmented" memory manager.  Virtual memory is
  allocated in segments, each of which may contain from 1K to 64K
  of data.  Initially, the VMM works somewhat like a normal
  memory allocator (although it has an extra advantage: most VM
  segments are movable; the VMM can reorganize them to get
  maximum utilization of real memory).  Unlike a normal memory
  allocator, however, the VMM does not "run out" when the
  available real memory can no longer contain all of the
  allocated segments.  Instead, one or more of the least recently
  used (LRU) segments are "swapped out" into secondary storage
  (or otherwise removed from real memory, see below) to make room
  for new segments.  Later, if data in a swapped segment must be
  accessed, it is swapped back in, displacing segments which are
  not currently needed.

  The maximum virtual address space of the Clipper 5.0 VMM is
  64MB.


USE OF LIM 3.2 EXPANDED MEMORY (EMM)

  By default, the Clipper 5.0 VMM uses LIM 3.2 Expanded Memory
  (EMM) as secondary storage.  EMM provides fast access to
  swapped segments.  Up to 8MB of EMM can be used.

  CONFIGURATION: EMM usage can be controlled using the E setting
  in the CLIPPER environment variable.  Valid settings are from 0
  to 8192 (units of 1K).  A setting of 0 disables the use of EMM
  altogether.  (Note: disabling EMM is not recommended under
  Clipper 5.0.  Although EMM is not required, it allows increased
  performance for most applications.  In a few cases, limiting
  EMM usage may be desirable -- see the note below under DISK
  BUFFERING SYSTEM.)

  Example:  SET CLIPPER=E2048

  CONFIGURATION: Some disk caching programs use EMM to cache disk
  sectors.  Certain cache programs may cause conflicts with the
  Clipper 5.0 VMM system if they assume that application programs
  never use EMM.  The BADCACHE setting in the CLIPPER environment
  variable causes the VMM to preserve and restore the state of
  the EMM "page frame" before and after every EMM access (the EMM
  page frame is an area in real address space through which EMM
  data is accessed).  This should make Clipper's use of EMM
  undetectable to any other process using EMM.  (Note: on some
  EMM systems the BADCACHE setting may adversely affect VMM
  performance.  It should only be used if you experience disk or
  file corruption because of a conflict with a disk cache or
  other resident software.)

  Example:  SET CLIPPER=BADCACHE


DISK SWAPPING

  If insufficient EMM is available to contain swapped segments, a
  temporary disk file (the "swap file") is created to hold them.
  By default, the VMM will use a maximum of 8MB of disk space for
  the swap file.  Disk space is used only as needed.

  CONFIGURATION:  The maximum size of the disk swap file can be
  controlled using the SWAPK setting in the CLIPPER environment
  variable.  Valid settings are from 256 to 64000 (256K to
  64MB).  The SWAPK setting has no effect on VMM swapping
  activity; it merely serves to establish an upper limit on the
  size of the swap file.  If the limit is exceeded, the VMM will
  terminate the application.

  Example:  SET CLIPPER=SWAPK:16000

  CONFIGURATION: The disk drive and directory where the VMM swap
  file is created can be controlled using the SWAPPATH setting in
  the CLIPPER environment variable.  By default, the swap file is
  created in the current DOS directory.

  Example:  SET CLIPPER=SWAPPATH:"C:\SWAP"


SWAP SPACE

  The term "swap space" refers to the region of real memory where
  the VMM loads virtual memory segments.

  CONFIGURATION: The VMM allocates the swap space from DOS at
  startup.  By default, all available DOS memory is used for swap
  space.  Real memory usage can be restricted using the X setting
  of the CLIPPER environment variable.  The setting specifies an
  amount of DOS memory (in 1K units) which is to be excluded from
  the VMM.  Valid settings are from 0 to 256.  The X setting is
  provided as a way to reserve DOS memory for other uses (e.g.
  allocation by memory-resident software).  In general, its use
  is undesirable because limiting the swap space decreases VMM
  performance.

  Example:  SET CLIPPER=X8


OBJECT MEMORY (SVOS)

  An "object memory" is a special type of memory manager designed
  to manage complex data values such as character strings and
  arrays.

  The Clipper 5.0 object memory is called the Segmented Virtual
  Object Store (SVOS).  SVOS uses virtual memory to store data
  values, including character strings, arrays, and dynamically
  created (macro-compiled) code blocks.

  SVOS provides two important functions beyond the basic
  capabilities offered by the VMM:

  - Memory compaction: stored values are automatically compacted
  on an ongoing basis.  This eliminates fragmentation of the
  virtual memory and reduces swapping.

  - Garbage collection: Some Clipper 5.0 values (e.g. arrays) may
  be referred to by several program variables (or array elements)
  at the same time.  The garbage collection algorithms
  automatically reclaim space occupied by "unreachable" values
  (values which are no longer accessible through any variable or
  array).

  To minimize delays associated with memory compaction and
  garbage collection, the SVOS algorithms are dynamically
  adjusted based on the type, size, and persistence of data being
  stored, the amount of memory available (real and EMM), and the
  overall performance of the system.  The algorithms attempt to
  manage memory in such a way that the VMM can swap out
  infrequently used data, freeing up real address space for
  time-critical operations.

  The dynamic nature of the SVOS algorithms makes it difficult to
  precisely state its maximum capacity.  The worst case capacity
  is in excess of 1MB.  The theoretical maximum is in excess of
  16MB.


DYNAMIC OVERLAYING

  Clipper 5.0, using the supplied special version of .RTLink,
  manages compiled Clipper code using a technique called dynamic
  overlaying.  This technique eliminates the need for complicated
  overlay structures and allows flexible runtime management of
  compiled code.

  During linking, .RTLink breaks compiled Clipper modules into
  fixed-size "pages."  These pages are stored either in the
  executable file or in separate overlay files.  The paging
  architecture eliminates restrictions and memory calculations
  based on the size of compiled functions or modules.  Large
  modules are broken into multiple pages; small functions are
  grouped together in a single page.

  At execution time, the dynamic overlay manager loads pages
  based on information embedded in the .EXE by the linker.  The
  dynamic pages are loaded into VM segments, allowing the VMM to
  manage the overlay pages on a competetive basis with other uses
  of memory.  The paging architecture allows the system to
  discard low-use sections of code even if the code is associated
  with a pending active function; overlays are not required to
  "nest" in parallel with function activations.  Code pages which
  are being heavily used are maintained in memory by the VMM's
  LRU swapping policy.

  When possible, the VMM will place dynamic overlay pages in EMM,
  reducing overlay reads.  Overlay pages are never written to the
  VMM disk swap file, however.  If a VM segment containing an
  overlay page is to be removed from memory altogether, it is
  simply discarded.  If it is needed subsequently, it is re-read
  from the overlay file.

  CONFIGURATION:  In addition to virtual memory, the dynamic
  overlay manager uses a dedicated area of real memory to cache
  the most active dynamic overlay pages.  The default size of
  this area ranges from 4K to 16K, depending on the amount of
  real memory available at startup.  For low memory situations,
  the size of this area can be restricted using the DYNK setting
  of the CLIPPER environment variable.  Valid values are from 4
  to 63.  (Note: restricting the size of this area may reduce
  execution speed.)

  Example:  SET CLIPPER=DYNK:4

  CONFIGURATION:  The dynamic overlay manager will hold overlay
  pages in virtual memory only as long as it can maintain an
  active file handle for the disk file containing the pages.  If
  the file is closed, the pages are discarded from virtual
  memory.  The maximum number of file handles used by the dynamic
  overlay manager can be controlled using the DYNF setting of the
  CLIPPER environment variable.  Valid settings are from 1 to 8.
  The default setting is 2.  For applications which use many
  separate overlay files, increasing this setting can improve
  performance.

  Example:  SET CLIPPER=DYNF:4


DATA FILE BUFFERING

  The Clipper 5.0 database drivers use virtual memory segments to
  contain database and index file buffers.  This allows the VMM
  to manage these buffers on a competetive basis with other uses
  of memory.  As with dynamic overlay segments, the VMM will swap
  buffer segments into EMM but not to the disk swap file.  If a
  VM segment containing a disk buffer is to be removed from
  memory altogether, it is written back to the appropriate data
  file.

  The file buffering system allocates buffer management tables in
  fixed memory (see FIXED MEMORY ALLOCATOR below).  The actual
  amount used depends on the combined total of real memory and
  EMM available at startup.  The tables use 14 bytes per 1K of
  real or Expanded memory (the tables never exceed 8K, however).
  Because these tables are allocated in fixed memory, they have
  an effect on the amount of swap space available to the VMM.  In
  some low-memory situations, reducing the amount of available
  EMM may help to avoid "swap space exhausted" errors. (Note:
  reducing EMM availability may seriously degrade VMM performance
  in very low memory situations due to increased disk swapping --
  see USE OF LIM 3.2 EXPANDED MEMORY above.)


SEGMENT LOCKING

  From time to time, certain VM segments are "locked" by a
  subsystem.  This prevents the VMM from moving or swapping those
  segments until the subsystem unlocks them.  This capability is
  used sparingly to keep the swap space from becoming cluttered
  with locked segments.


FIXED MEMORY ALLOCATOR

  Clipper 5.0 subsystems use virtual memory to contain code or
  data that needs to be dynamically managed.  Some allocations
  are not appropriate for VM (e.g. system tables which must
  remain in the same location throughout execution).  A C-style
  fixed memory allocator is included for this purpose.  The fixed
  memory system allocates one or more VM segments and permanently
  locks them in real memory.  Space within these segments is then
  parceled out as needed.  Since fixed segments are never
  unlocked, the VMM locates them at the low end of the swap space
  to prevent them from interfering with other operations.


MEMVAR TABLES

  Clipper 5.0 offers several different storage classes for
  program variables.  Local and static variables are stored in a
  dedicated area of real memory.  Private and public (MEMVAR)
  variables are stored in VM segments.  Since MEMVAR variables
  are created and destroyed dynamically during execution, the
  associated VM segments grow dynamically depending on how many
  MEMVAR variables are in use.  For performance reasons, these
  segments remain locked during most operations (they are
  unlocked during RUN commands and certain other memory intensive
  operations).  In low memory situations, the MEMVAR tables may
  have a significant effect on the amount of available swap space
  (each MEMVAR variable uses 20 bytes in the MEMVAR segments,
  which grow in increments of 1K or more).  MEMVAR variables also
  require symbol information to be maintained at runtime (see
  below).  Converting private and public variables to local and
  static variables may reduce memory requirements for some
  applications.


SYMBOL TABLES

  Clipper maintains symbol information at runtime in order to
  support symbolic references to variables and functions.  Some
  symbol information is embedded in the .EXE file (the Clipper
  5.0 version of .RTLink eliminates duplication among symbols
  from compiled Clipper modules).  Other symbols are created
  dynamically at runtime.  Memory for symbol tables and related
  information is allocated using the fixed memory allocator.  The
  symbols are allocated in blocks and do not fragment memory.
  However, excessive dynamic symbol creation (e.g. large numbers
  of MEMVAR variables created via macros) may cause the symbol
  tables to grow, reducing swap space.


THE EXTEND SYSTEM

  The Extend system is a set of functions that are used by C and
  assembly language programs to interface with Clipper programs.
  The Extend system includes two memory allocation functions,
  _xalloc() and _xgrab(), which allow C programs to allocate
  memory from Clipper.  These functions call the fixed memory
  allocator (see above).

  The Extend system also allows C programs to gain access to
  Clipper data values.  The Extend system _parc() function, used
  to inspect character values passed from Clipper, is defined as
  returning an address in real memory.  Since character values
  are stored in virtual memory under SVOS (see above), the Extend
  system automatically locks the appropriate VM segments in
  response to _parc() requests.  The segments are automatically
  unlocked when the C function returns control to Clipper (they
  may also be unlocked manually from C, see below).

  A conflict can occur if a C function locks a VM segment via
  _parc() and then attempts to allocate a large amount of memory
  using the Extend allocators.  If the segment is locked near the
  low end of the swap space, the fixed allocator may be unable to
  lock or expand a fixed segment to satisfy the allocation
  request.  If this occurs, a new fixed segment is created and
  locked higher in memory.  This reduces memory efficiency
  because the VMM is not allowed to move fixed segments (in
  severe cases, the VMM may issue a "swap space exhausted" error
  because the swap space has become too cluttered with fixed
  segments).  To prevent this, the VMM maintains a "safety zone"
  at the low end of the swap space.  _parc() requests are
  prevented from locking segments within this zone.

  CONFIGURATION: The size of the Extend "safety zone" can be
  adjusted using the EXTHEAP setting of the CLIPPER environment
  variable.  The default setting is 8K.  Valid settings are from
  2 to 256 (units of 1K).  Increasing the safety zone allows C
  programs to allocate large amounts of memory without creating
  "holes" in the swap space.  (Note: the default EXTHEAP setting
  should be sufficient for most applications; increasing the
  value too much may cause the VMM to fail when trying to lock a
  segment in response to a _parc() request.)

  NEW EXTEND FUNCTION: The Clipper 5.0 VMM defines an entry point
  called _xunlock().  This function can be called from C or
  assembly language programs to unlock SVOS segments when large
  amounts of memory are to be allocated after _parc() has been
  called.  CAUTION: calling _xunlock() causes any pointers
  returned from previous _parc() calls to become invalid.  To
  access a Clipper character value after calling _xunlock(), you
  must make another call to _parc() to get a new pointer.

  Nantucket will be publishing an API (Application Program
  Interface) for the Clipper 5.0 VMM so that C and assembly
  language programs can allocate virtual memory directly,
  reducing demands on swap space.


END: CLIPPER 5.0 SUPPORT BULLETIN #3